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ABSTRACT* 

Objectives: To develop and study the validity of an 
instrument for evaluation of Printed Education 
Materials (PEM); to evaluate the use of acceptability 
indices; to identify possible influences of 
professional aspects. 

Methods: An instrument for PEM evaluation was 
developed which included tree steps: domain 
identification, item generation and instrument 
design. A reading to easy PEM was developed for 
education of patient with systemic hypertension and 
its treatment with hydrochlorothiazide. Construct 
validity was measured based on previously 
established errors purposively introduced into the 
PEM, which served as extreme groups. An 
acceptability index was applied taking into account 
the rate of professionals who should approve each 
item. Participants were 10 physicians (9 men) and 5 
nurses (all women). 

Results: Many professionals identified intentional 
errors of crude character. Few participants identified 
errors that needed more careful evaluation, and no 
one detected the intentional error that required 
literature analysis. Physicians considered as 
acceptable 95.8% of the items of the PEM, and 
nurses 29.2%. The differences between the scoring 
were statistically significant in 27% of the items. In 
the overall evaluation, 66.6% were considered as 
acceptable. The analysis of each item revealed a 
behavioral pattern for each professional group. 
Conclusions: The use of instruments for evaluation 
of printed education materials is required and may 
improve the quality of the PEM available for the 
patients. Not always are the acceptability indices 
totally correct or represent high quality of 
information. The professional experience, the 
practice pattern, and perhaps the gendre of the 
reviewers may influence their evaluation. An 
analysis of the PEM by professionals in 
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communication, in drug information, and patients 
should be carried out to improve the quality of the 
proposed material. 
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RESUMEN 

Objetivos: Desarrollar y estudiar la validez de un 
instrumento para la evaluacion del material 
educativo impreso (MEI); evaluar el uso de los 
indices de aceptabilidad; identificar las posibles 
influencias de los aspectos profesionales. 
Metodos: Se desarrollo un instrumento para 
evaluacion del MEI en tres pasos: identification de 
dominios, generation de items y diseno del 
instrumento. Se desarrollo un MEI facil de leer para 
la education de pacientes con hipertension 
sistemica y su tratamiento con hidroclorotiazida. Se 
midio la validez del instrumento mediante los 
errores previamente introducidos a proposito en el 
MEI, lo que sirvio de grupo extreme Se aplico un 
indice de aceptabilidad teniendo en cuenta la tasa 
de profesionales que tenia que aprobar cada item. 
Participaron 10 medicos (9 hombres) y 5 
enfermeras (todas mujeres). 

Resultados: Muchos profesionales identificaron los 
errores intencionales de caracter crudo. Pocos 
participantes identificaron los errores que 
necesitaban una evaluacion mas cuidadosa, y 
ninguno identified el error intencional que requeria 
analisis de la literatura. Los medicos consideraron 
aceptable el 95,8% de los items del MEI y las 
enfermeras el 29,2%. Las diferencias de puntuacion 
fueron estadisticamente significativas en el 27% de 
los items. En la evaluacion total, el 66,6% fue 
considerado aceptable. El analisis de cada item 
revelo un modelo de comportamiento para cada 
grupo de profesionales. 

Conclusiones: Es necesario el uso de instrumentos 
para la evaluacion de materiales educativos 
impresos y puede mejorar la calidad de los MEI 
disponibles para los pacientes. Los indices de 
aceptabilidad no son siempre totalmente correctos 
ni representan la calida de la information. La 
experiencia profesional, el modelo de ejercicio, y 
quizas el genero de los revisores puede influir en la 
evaluacion. Deberia realizarse un analisis de los 
MEI por profesionales de comunicacion, de la 
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information sobre medicamentos y por pacientes 
para mejorar la calidad del material propuesto. 

Palabras clave: Educacion de pacientes. 
Comprension. Estudios de validation. Brasil. 



(English) 



INTRODUCTION 

Printed education material (PEM) is one of the most 
frequently used resources for the education of 
patients with chronic diseases. Different authors 
have established the most important steps in 
developing PEM 1 " 10 for literate patients. Some 
articles in the current literature have reported the 
poor quality of information of such materials 11 " 16 , as 
well as the lack of scientific accuracy to promote the 
education of patients. 11 ' 12 ' 16 " 23 

Printed education materials have been used 
routinely in Brazil for patients from the public and 
private institutions, but with no defined validation 
process. Two reports on evaluation of such 
materials for Brazilian patients were found. 24 ' 25 The 
first focuses on criteria for evaluation of textbooks 
for elementary education 24 and the second 
evaluates the acceptability of one piece of 
education material produced in the form of a 
game. 25 The lack of assessment of PEM produced 
in the Portuguese language and the new 
requirements for the development of PEM 17,23 
demonstrate that it is necessary to develop further 
research in the educational area. In this report, the 
development and testing of an instrument for 
evaluation of PEM (EVALPEM) for health 
professionals and the performance of the 
acceptability index of the evaluated items, and the 
influence of some professional aspects in this 
process are presented. 



METHODS 

Development of the evaluation tool 

The steps proposed by Lynn 2 were used for 
identification of domains, generation of items and 
tool design. 

Phase 1 : Identification of domains 

Domains were those identified in the literature 
review or regarded as important for determining the 
quality of a PEM, according to expert 
evaluation. 3 ' 17 ' 23 ' 26 

Phase 2: Items Generation 

The items were established for each domain 
(described in Table 1), specifying the basic 
principles pertinent to each item. 3,16 ' 17 ' 26 

Phase 3: Design of the Instrument 

Based on the previously established domains and 
their respective items, a tool for the evaluation of 
PEM was developed. Each item was presented in 
the form of an affirmation, followed by an 
assessment scale with 3 categories: totally 
disagree, corresponding to 0 points; partially agree, 
corresponding to 1 point; totally agree, 
corresponding to 2 points. This scale was adapted 
from the scale proposed in The Bernier Instructional 
Design Scale (BIDS). 3 After the scoring scale, there 
was a blank space for comments. In the 
instructions, the evaluators were asked to report 
any problems faced in this field while setting up the 
PEM or to state "not applicable" when appropriate. 
At the end of the EVALPEM, four questions were 
asked, aiming an integral evaluation of the PEM: 1) 
What did you like in the PEM? 2) What did you not 
like in the PEM? 3) What should be added? 4) What 
should be reviewed? 



Table 1 - Part 1 : Principles and criteria for evaluation of Printed Educational Materials. 

Principle 

1 - Scientific Accuracy 

a) contents are in agreement with the current knowledge 

b) recommendations are necessary and are correctly approached 

2 - Content 

a) objectives are evident 

b) recommendation about the desired behavior is satisfactory 

c) there is no unnecessary information 

d) important points are reviewed 

3 - Literary Presentation 

a) language is neutral (no comparative adjectives, promotion or false appeals) 

b) language is explanatory 

c) language is conversational and, in at least 50% of the material, written in the active voice 

d) material promotes and encourages treatment adherence following evaluation of the benefits and risks 

e) majority of the vocabulary is composed of common words 

f) context of each issue is communicated before the new information 

g) identification of headings and subheadings help in the learning process 

h) vocabulary is composed of simple words 

i) language is adequate for outpatients 

j) ideas are concisely expressed 

k) text allows for interaction with verbal counseling 

I) text allows for interaction with logical linkage of the therapeutic plan 

m) planning and sequence of information is consistent, making it easier for the patient to predict its flow 
n) material is reader-friendly 
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Table 1 - Part 2: Principles and criteria for evaluation of Printed Educational Materials. 

Principle 

4 - Illustrations 

a) illustrations are simple, appropriate and present an easily understandable outline 

b) they are familiar to the readers 

c) they are related to the text (express the desired purpose) 

d) they are integrated with the text (easily located) 

e) lists, tables and graphs are self explanatory 

f) headings in e) are adequate 

5 -Material is sufficiently specific and understandable 

a) material promotes the correct use of the medication 

b) provides maximum benefit with minimization of complications 

c) instructions for administration of treatment are clear 

d) recommendations on how to prevent complications are understandable 

e) it is clearly explained how to identify the action of drug 

f) material enables reader to recognize when they have to urgently look for a physician assistance 

g) enables the patient to determine whether he or she is facing a serious problem 

h) technical terminology is adequately defined 

i) headings and subheadings are clear and informative 

j) use of words or expressions with double meanings does not occur in the text 

k) content is written in a patient-centered style; that is, the patient is the focus of importance 



Table 1 - Part 3: Principles and criteria for evaluation of Printed Educational Materials. 

Principle 

6 - Legibility and Printing Characteristics 

a) size of the letters is adequate 

b) style of the letters is adequate 

c) spacing between letters is adequate 

d) length of the lines is adequate 

e) spacing between lines is adequate 

f) use of bold characters and bullet points draw attention to specific points or key content 

g) adequate use of blank space reduces overcrowded appearance 

h) good contrast between the printed content and the paper 

i) paper used makes it easier to read 

j) subheadings or the inner margins make reading and memorization easier 

k) spacing between paragraphs is adequate 

I) format of the material is adequate 

7 - Quality of Information 

a) it is integrated to the local culture 

b) information is updated 

c) it is adapted to the current culture 

d) material enables the patient to undertake the desired actions 

e) material helps patient to prevent potential problems 

f) material allows patient to achieve the maximum benefits possible 



Construct validity 

It was not possible to validate EVALPEM as 
proposed by Bernier 3 and Coulter et al. 17 since it 
was not possible to find a sufficient number of 
experts in patient education in Brazil. Therefore, we 
decided to evaluate the validity of the designed 
instrument using an alternative method. The 
introduction of errors previously established and 
introduced in a PEM constructed for this purpose 
was the artifice used, thus providing correct and 
incorrect elements that acted as extreme groups. 27 
Through the analysis of the "extreme groups," we 
intended to verify whether the evaluators would be 
capable of detecting errors using the proposed 
instrument, as well as identifying possible 
professional influences when applying a tool like 
EVALPEM. 

The PEM consisted of two blocks, one related to 
systemic arterial hypertension and the other to the 
use of hydrochlorothiazide. These themes were 
chosen due to the high prevalence of hypertension 



in our city 1 and the use of hydrochlorothiazide as 
one of the first choice medications for the 
management of this condition. 29 " 31 A version which 
required less than 6 years of schooling from 
patients was utilized. 32 The basic structure of the 
PEM was previously presented. 17,33 

Procedure for application of the EVALPEM 

Health professionals that provided outpatient 
services in the Hospital de Clinicas de Porto Alegre 
and who were involved in the care of patients with 
hypertension were invited to evaluate the PEM. All 
of them agreed to participate of the study, not taking 
into account their real objectives or the existence of 
extreme groups. 

For analytical purposes, they were divided into 2 
groups. The first (Group 1) consisted of 10 
physicians (9 men) from the Division of Cardiology. 
The second (Group 2) consisted of 5 nurses (all 
women) of the Nursing Service in Public Health. 
They received a copy of the PEM and the 
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evaluation tool, as well as written instructions for 
completing it. After one week, the materials were 
returned with the questionnaires completed. The 
data from the two groups of professionals were 
analyzed separately and jointly in order to identify 
differences associated with their background. 

Evaluation of the acceptability index of the 
proposed material 

To evaluate the contents of the PEM, the 
parameters described by Lynn 2 were adopted. 
These parameters state that the acceptability of a 
given criterion depends on the approval of a certain 
proportion of experts. This proportion varies 
according to the numbers of evaluators. In Group 1 , 
an item was considered acceptable when at least 
80% of the possible points were obtained; in Group 
2, 100% of the points were required to approve the 
item. The score for both groups combined was set 
in 80%. 

RESULTS 

In the process of construct validity, the physician 
and the nursing groups identified, respectively, 6 
and 7 from 9 intentional errors. For two items only 
one professional identified the errors: 1) the 
absence of report concerning the adequate arm 
position in the measurement of blood pressure, and 
2) the inadequate spacing between lines. The 
intentionally false contraindication of 
hydrochlorothiazide in patients with rheumatoid 
arthritis was not identified by any professional. 

Based on the previously established parameters, it 
was observed that for the physician group 95.8% of 
the items in the PEM were considered acceptable. 
Six items were considered unacceptable. Four of 
them corresponded to the errors deliberately 
introduced in the PEM. The 2 other corresponded to 
the items that physicians did not either totally or 
partially agree with. The description of the 
mechanism of the action of the drug was considered 
difficult for patients to understand, and the 
description of adverse effects of hydrochlorothiazide 
was considered unnecessary. 

For the nurse group, just 29.2% of the items were 
considered as acceptable. Forty items were 
considered as unacceptable, 7 of them referring to 
the deliberate errors that were adequately identified. 
In relation to the others, a behavior similar to the 
physicians was observed in regard to the amount of 
information to be provided to the patient. Moreover, 
nurses were more critical, indicating inadequacies in 
the PEM that were not reported by the physicians, 
such as double meaning expressions and usage of 
terminology considered difficult to understand by the 
patients. 

The differences between the scoring of the 2 groups 
were statistically significant in 27% of the items (P < 
0.05, Mann-Whitney U test). In the overall 
evaluation, 66.6% of the items were considered as 
acceptable. 

The analysis of each item revealed a behavioral 
pattern for each professional group. Among the 
physicians, 7 suggested modifications in the PEM, 



such as to use this opportunity to better explain the 
risks for arteriosclerosis, to teach how to recognize 
angina and stroke. One physician suggested an 
investigation of patients in order to determine the 
best way to orient them about the administration of 
hydrochlorothiazide. Five physicians were against at 
least one of the following items: to inform patients 
about the stages of hypertension; to measure BP 
outside the medical office; to inform about potential 
adverse drug reactions; to require the control of 
blood glucose before starting the use of 
hydrochlorothiazide. 

Four nurses gave suggestions: to extend the 
concept of hypertension; to highlight titles in a 
different color from the printed text; to identify the 
producers and reviewers in the PEM; and to better 
explain the relationship between loss of potassium 
and cramps. They also suggested that some points 
should be directly investigated with the patients. 
Four nurses were against at least one of 8 points, 
for example, the explanation of the classification of 
hypertension and the orientation on how to prevent 
intercurrences, since they might scare patients and 
reduce concordance of the treatment. 

Some comments from the nurses were in 
disagreement with the recommendations from the 
literature, such as (1) disagreement regarding to 
diet orientation; (2) the possibility of controlled 
alcohol and salt consumption, opting for prohibiting; 
(3) the explanation on how to conduct in face of lack 
of one dose of the drug, since it could lead to 
confusion, and (4) the risk of photo-sensitivity 
associated with the use of hydrochlorothiazide. 

Contrasting ideas on the same topic were also 
observed. A physician considered the material too 
simple while 3 nurses stated that the material was 
too long. Three physicians and a nurse reported 
their concern about the participation of pharmacists 
in the education of patients. Three physicians and 
four nurses did not agree to warn patients about 
potential adverse reactions to hydrochlorothiazide 
and deemed unnecessary for patients to know the 
stages of hypertension. 

DISCUSSION 

The quality of information provided to patients 
unequivocally contributes to the prevention of 
illnesses and recovery of health. 34 " 36 The quality of 
such materials should be evaluated. Previously to 
their use by patients, PEM needs to be evaluated by 
different groups of professionals involved with their 
care. Therefore, the development of instruments for 
this evaluation is required, such as the presented 
and tested in this report. 

The questionnaire applied in this study was detailed 
and some items were considered irrelevant by the 
professionals. Both groups identified correctly the 
most evident deliberate errors. However, just a few 
professionals were able to identify the errors that 
would require a deeper analysis of the sources of 
information or about legibility and printing 
characteristics. These findings suggest that, in the 
domain of scientific accuracy, experts in the 
analysis of scientific literature should also review 
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PEM information, as recommended by Coulter. 17 
Additionally, professionals of the communication 
area should be consulted about the illustrations, 
legibility and printing characteristics of the material. 

The EVALPEM, which was tested in the evaluation 
of the printed educational material on hypertension 
and the use of hydrochlorothiazide, allowed 
identifying the opinions of physicians and nurses 
about the PEM. The physicians accepted most part 
of what was reported in the PEM, while the nurses 
did not accept many items. 

The items that have discordant opinions from these 
two groups should be studied. Part of the different 
evaluation by physicians and nurses may be 
attributed to the different knowledge about the 
theoretical background topic by the professionals. 
Physicians were in charge of an outpatient clinic of 
hypertension, while the nurses that were invited to 
participate came from the Nursing Service in Public 
Health Hospital. We observed that much of the 
disagreement regarding the proposed PEM derive 
from their models of professional practice (The 
Medical Model versus The Helping Process) 34 and 
from the clinical decision making. 17 Professionals 
tend to establish a relation of dependence rather 
than partnership with their patients. Oliveira 
reported similar results 37 in a study about the 
communication process between patients and 
physicians when a serious prognosis should be 
given to patients. Differences related to the gender 
could not be also discarded, since all nurses were 
women and most physicians were male. 

We suggest that isolated remarks should not be 
discarded. Sometimes just one individual perceives 
the problems regarding to one topic, while all other 
evaluators approve it. These data reveal the 
limitation of using isolated acceptability indices in 
the PEM evaluation. 



On the other hand, the fact of having 
consensus among the professionals on a particular 
item does not mean necessarily adequacy. Both 
groups, for example, disagreed on presenting 
potential drug adverse effects, a recommendation 
not supported by the best evidence. 38 " 43 The PEM 
improved on the basis of the evaluation by 
physicians and nurses should be sequentially 
evaluated by professionals of communication and 
by the patients themselves. 

The reaction to the participation of pharmacists in 
patient education expressed by some physicians 
and nurses probably reflects the model of 
professional pharmaceutical practice adopted in 
Brazil, which is primarily oriented to drug 
distribution. The adoption of a new paradigm - 
pharmaceutical care - one of the World Health 
Organization recommendations 44 ' 45 , might 
contribute to solving health issues related to the use 
of medicines. 

CONCLUSION 

The present study demonstrated that printed 
education materials require the use of an evaluation 
process to improve their quality. Variables such as 
the evaluation instrument, the profile of reviewers, 
and the acceptability index should be used 
appropriately. 
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